Image processing system for estimating the energy transfer of an occupant into an airbag

ABSTRACT

The present invention relates in general to systems used to determine whether an airbag should be deployed at full or only partial strength. In particular, the present invention is an image processing system that utilizes real-time streaming video-images from a video camera or other sensor to determine the mass, velocity, and kinetic energy of the occupant at the time that the occupant comes into contact with the deploying airbag. By predicting the kinetic energy of the occupant at the time of impact, an airbag can be deployed at an appropriate strength corresponding to the kinetic energy of the occupant. The kinetic energy of the deploying back at the moment of impact should be equal to the kinetic energy of the occupant. The invention captures the volume of the occupant from an image, and uses volume to calculate the mass of the occupant. A Kalman filter is used with respect to all measurements to incorporate past predictions and measurements into the most recent estimates and predictions in order to eliminate the “noise” associated with any particular measurement. The system predicts the position and shape of the occupant at a faster rate than the rate at which the camera collects data.

RELATED APPLICATIONS

[0001] This Continuation-In-Part application claims the benefit of theU.S. utility application titled “IMAGE PROCESSING SYSTEM FOR DYNAMICSUPPRESSION OF AIRBAGS USING MULTIPLE MODEL LIKELIHOODS TO INFER THREEDIMENSIONAL INFORMATION,” Ser. No. 09/870,151, filed on Jul. 10, 2001,the contents of which are hereby by incorporated by reference in theirentirety.

BACKGROUND OF THE INVENTION

[0002] The present invention relates in general to systems or methodsused to determine whether the deployment of an airbag should be eitherfully or partially impeded. In particular, the present invention is animage processing system that receives a stream of two-dimensional imagesand applies iterative and probability-weighted processes to inferthree-dimensional characteristics to those images, which in turn areused to determine the magnitude of the impact between an occupant and anairbag.

[0003] Conventional airbag deployment systems have contributedsignificantly to the safety of occupants in automobile crashes. However,there may be occasions when the full strength deployment of an airbagmay not be desirable. The purpose of a airbag is to absorb the impact ofan occupant. Airbag deployment strength in excess of this purpose maynot be desirable.

[0004] Prior art systems generally take an all or nothing approach.Under such approaches, an airbag is either fully disabled or is deployedat full strength. A total disablement of the airbag precludes theoccupant from receiving any benefits of an airbag, a generally usefulsafety device. A full strength deployment of an airbag may subject theoccupant to undesirable force in a low speed crash. It would bedesirable if an airbag deployment system could be deployed at variousdifferent strengths depending on the impact of the occupant and theairbag. It would be desirable if metrics such as kinetic energy,momentum, or other measurements utilizing the characteristics of massand velocity were used to determine the magnitude of impact that anairbag needs to absorb from the impacting occupant. Current methods fordetermining the velocity and mass of an occupant suffer from significantlimitations in the existing art.

[0005] Weight-based systems typically use weight sensors to determinethe weight of an occupant. Weight-based systems are vulnerable toinaccurate measurements because of rapidly changing movements of anautomobile in a state of crashing. Such movements can cause the weightin a seat to shift, making it difficult to measure weight accurately.Weight-based systems also lack the ability to track the speed at whichthe occupant approaches the airbag during a crash or during pre-crashbraking. It would be desirable if the weight of an occupant weremeasured by more reliable means, such as by an analysis of the visualimage of the occupant. Because the weight density for the otherwisediverse range of human occupants is relatively constant, it would bedesirable to use the volume of an occupant to determine the mass of theoccupant. It would also be desirable for the velocity of the occupant tobe measured or calculated.

[0006] The current art also suffers from problems relating to thedetermination of velocity. Velocity-based systems often require highlyexpensive cameras. Timing is critical to any system used to modifyairbag deployment. A standard video camera operates at a frequencybetween 50-100 hz and captures between 50 to 100 image frames per secondof operation. Effective airbag determinations require more frequentupdates, around approximately 200 updates per second (200 hz). Moreover,it would be desirable for an image processing system to predict theoccupant's velocity at the time of impact with the airbag rather thanmerely identifying the occupant's velocity at the time that the image iscaptured. It would be desirable for such a system to track theacceleration and location of the occupant. It would also be desirable ifaccurate predictions could be generated at a faster rate than the cameraspeed so that a less expansive standard video camera could be usedinstead of a more expensive highly specialized high-speed camera.

[0007] Prior art systems are susceptible to “noise” because prior artsystems focus solely on the most recent measurement or image, and ignorethe series of measurements or images captured mere fractions of a secondearlier. Measurement “noise” results from several factors, including theinherent imperfections of the segmentation process. The segmentationprocess is the process of extracting a segmented image (an image of justthe occupant in isolation of the surrounding area) from the ambientimage, which includes the image of the occupant as well as thesurrounding area. It would be desirable for an image processing systemto utilize an iterative process that would integrate the informationcontained in the most recent image into a comprehensive framework thatincludes prior predictions and indirectly, the prior images used to makethose prior predictions. It would also be desirable for all predictionsto be probability-weighted predictions. It would also be desirable forsuch weighted predictions to incorporate probabilities associated withpredefined occupant states such as leaning left towards the driver,leaning right away from the driver, or sitting upright, and predefinedoccupant modes such as crash, stationary, or human.

[0008] Camera-based prior art systems also limited by the fact that theyrely on two-dimensional images. The images captured by cameras,including video cameras, are inherently two dimensional images. However,some important characteristics such as volume are three dimensional.Thus, it would be highly beneficial if three-dimensional informationcould be inferred from a series of two-dimensional images taken by asingle standard video camera. Moreover, it would be helpful ifpredefined occupant states were incorporated into the iterative processof deriving a three-dimensional information from a series oftwo-dimensional images.

SUMMARY OF THE INVENTION

[0009] This invention relates to an image processing system used todetermine the strength at which an airbag should be deployed. Inparticular, this invention relates to an image processing system used todetermine the impact of the occupant that the deploying airbag needs toabsorb.

[0010] The invention determines the kinetic energy, momentum, or someother metric for measuring the impact of the occupant, from an image ofthe occupant. The impact measuring metric can then be used by the airbagdeployment system to determine the desirable strength for airbagdeployment. For example, the impact metric may indicate that an airbagshould only be deployed at 100%, 75%, 50%, or even only 25% strength.The impact metric could also determine that an airbag should not deployat all (a 0% strength deployment).

[0011] Through the use of sensor readings, the invention calculates oneor more impact metrics. In a preferred embodiment of the invention,kinetic energy is the impact measurement. Kinetic energy is calculatedusing the mass and velocity of the occupant. Mass can be calculatedusing the volume of an occupant. Velocity of an occupant can bedetermined by comparing the change in occupant position.

[0012] In a preferred embodiment of the invention, two iterative andinteractive multiple model Kalman filters can be used to incorporate aseries of measurements relating to the most recently captured image ofthe occupant into an ongoing series of past predictions and imagemeasurements relating to the position (including position, velocity, andacceleration) and shape (including area and volume) of the occupant.

[0013] Various aspects of this invention will become apparent to thoseskilled in the art from the following detailed description of thepreferred embodiment, when read in light of the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014]FIG. 1 shows a partial view of the surrounding environment for onepotential embodiment of the invention.

[0015]FIG. 2 shows a high-level process flow of the image processingsystem.

[0016]FIG. 3 shows a high-level block diagram of an image processingsystem.

[0017]FIG. 4 shows the results of an ellipse fitting routing asperformed by the ellipse fitting subsystem.

[0018]FIG. 5 shows an upper torso ellipse with all corresponding shapevariables and motion variables (excluding velocity and accelerationderivations).

[0019]FIG. 6 shows three potential shape states and three potentialsideways tilt angles.

[0020]FIG. 7 shows a markov chain of probabilities relating to a threeshape state embodiment including left, right, and center states.

[0021]FIG. 8 shows a markov chain of probabilities relating to a threemotion mode embodiment including of human, stationary, and crash modes.

[0022]FIG. 9 shows a detailed flowchart for determining the shape of theoccupant.

[0023]FIG. 10 shows a detailed flowchart for determining the motion ofthe occupant.

[0024]FIG. 11a shows a “look-up” table where the width of the upperellipse is determined by using the major axis and minor axis of theupper ellipse as inputs.

[0025]FIG. 11b shows a “look-up” table where the volume of the upperellipse is determined by using the major axis and minor axis of theupper ellipse as inputs.

[0026]FIG. 11c shows a “look-up” table where the mass of the upperellipse is determined by using the major axis and minor axis of theupper ellipse as inputs.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A. Partial View ofSurrounding Environment

[0027] Referring now to the drawings, illustrated in FIG. 1 is a partialview of the surrounding environment for one potential embodiment of theinvention, an image processing system 16. If an occupant 18 is present,the occupant 18 sits on a seat 20. In the preferred embodiment, a cameraor other sensor 22 (collectively “camera”) capable of rapidly capturinga series of images is attached in a roof liner 24, above the occupant 18and in a position closer to a front windshield 26 than the occupant 18.The camera 22 should be placed in a slightly downward angle towards theoccupant 18 in order to capture changes in the angle of the occupant's18 upper torso resulting from forward or backward movement in the seat20. There are many potential locations for a camera 22 that are wellknown in the art. Similarly, a computer 30 could be located virtuallyanywhere on a vehicle. In a preferred embodiment, the computer 30 islocated near the camera 22 to avoid sending camera images through longwires. An airbag controller 32 is shown in an instrument panel 34,although the airbag controller could be located virtually anywhere inthe vehicle. An airbag deployment system 36 is shown in the instrumentpanel 34 in front of the occupant 18 and the seat 20, although thesystem 16 can function with the airbag deployment system 36 inalternative locations.

B. High Level Process Flow

[0028]FIG. 2 discloses a very basic and high-level process flow relatingto the use of the image processing system 16. An ambient image 38 of aseat area 21 includes both the occupant 18 and the surrounding seat area21. In contrast to the ambient image 38 is a segmented image, whichincludes only the image of the occupant 18 in isolation from thesurrounding seat area 21.

[0029] The ambient image 38 is captured by the camera 22 or any othersensor capable of rapidly capturing a series of images. In the Figure,the seat area 21 includes the entire occupant, although under somecircumstances and embodiments, only a portion of the occupant's 18 imagewill be captured, particularly if the camera 22 is positioned in alocation where the lower extremities may not be viewable. The ambientimage 38 is sent to the computer 30.

[0030] As described in greater detail below, the system 16 derives asegmented image from the ambient image 38. The system 16 captures aseries of measurements and performs a series of calculations todetermine the magnitude of an impact metric between the occupant 18 andthe airbag that will need to be absorbed at the time of impact.

[0031] In the preferred embodiment, the impact metric used by the system16 is the kinetic energy of the occupant 18. In alternative embodiments,the momentum of the occupant 18, a weighted combination of kineticenergy or momentum, or any other heuristic or combination of heuristicsutilizing mass and velocity can be used as the impact metric. The impactmetric is sent to the airbag controller 32 which in turn controls theairbag deployment system 36. The airbag deployment system 32 uses theimpact metric to determine the impact of the occupant 18 that needs tobe absorbed by the airbag. The airbag controller 32 can then cause theairbag deployment system 36 to deploy the airbag at the appropriatestrength, such as 100%, 75%, 50%, 25%, or even 0% (no deployment atall).

C. Computer System Process Flow

[0032]FIG. 3 gives a high level process flow as to the processingperformed by the computer 30 in the various embodiments of the system16. In a preferred embodiment, the ambient image 38 of the occupant 18and surrounding seat area 21 is inputted to the computer 30. Asegmentation subsystem 40 extracts a segmented image 42 of the occupant,an image which includes only the occupant and not the surrounding seatarea 21, from the ambient image 38. Various segmentation processes 40are well known under existing art. Segmentation 40 is accomplishedelectronically through the use of software, and thus in alternativeembodiments of the invention, segmentation could be performed in acomputer that is separate from the computer at 30.

[0033] The segmented image 42 of the occupant 18 is then processed by anellipse fitting subsystem 44. An upper ellipse is fitted around theupper torso of the occupant 18. The upper torso includes everything fromthe head of the occupant 18 down to the hips of the occupant 18. A lowerellipse is fitted from the toes of the occupant 18 up to the hips of theoccupant. Both ellipses overlap at one point which is referred to as thecentroid. The output of the ellipse fitting routine 44 will be discussedin greater detail below along with the discussion relating to FIGS. 4,5, and 6. The processes and practices of generating an upper ellipse, alower ellipse, and a centroid to represent different parts of theoccupant 18 are known in the art. In a preferred embodiment of theinvention, the ellipse fitting routine 44 is applied by the computer at30, but in alternative embodiments, the ellipse fitting subsystem 44could be performed by a computer that is separate from but connected tothe computer at 30. Alternative embodiments could also utilize adifferent geometric shape than an ellipse, such as a rectangle, circle,or other geometric shape.

[0034] Information relating to the upper ellipse, lower ellipse, andcentroid are then sent to a tracking and predicting subsystem 46. Thetracking and predicting subsystem 46 applies distinct sub-processessimultaneously with each other. A shape tracker 48 tracks variablesrelating to the “shape” of the upper ellipse. This process is describedin greater detail below in the discussion relating to FIG. 9. A motiontracker 50 tracks variables relating to the “motion” of the upperellipse, specifically with respect the horizontal distance between theoccupant 18 and the airbag deployment system 36 in the instrument panel34. This process is described in greater detail below in the discussionrelating to FIG. 10.

[0035] The computer system 30 uses the tracking and predicting subsystem46 to track and predict the position, velocity, and acceleration ofcertain key measurements relating to shape and motion of the upper torsoof the occupant 18 using a multiple model and iterative process. Thisiterative process incorporates the latest measurement into a series ofongoing measurements and predictions. In a preferred embodiment, thetracking and predicting subsystem 46 incorporates what is known in theprior art as a Kalman filter. A Kalman filter is very powerful in thatit incorporates past predictions and measurements in a weighted fashion,supports estimations of past, present, and even future states. A Kalmanfilter can be used effectively even when the precise nature of themodeled system is unknown.

[0036] An academic paper entitled “An Introduction to the Kalman Filter”by Greg Welch and Gary Bishop is attached and incorporated by reference.The general equation for the Kalman filter is shown in Equation 1:

X _((new prediction)) =X _((old prediction))+Gain[−X _((old prediction))+X _((measured))]

[0037] In a Kalman filter, “Gain” represents the perceived accuracy ofthe most recent measurement. A Gain of 0 indicates such a poormeasurement that it is of no value, and thus the new estimateX_((new estimate)) is simply the value of the old estimateX_((old estimate).)

X _((new estimate)) =X _((old estimate))+0[−X _((old estimate)) +X_((measured))]

X _((new estimate)) =X _((old estimate))+0

X _((new estimate)) =X _((old estimate))  Equation 2

[0038] A Gain of 1 indicates such confidence in the most recentmeasurement X_((measured)) that the new prediction X_((new estimate)) issimply the value of the most recent measurement X_((measured).)

X _((new estimate)) =X _((old estimate))+1[−X _((old estimate)) +X_((measured))]

X _((new estimate)) =X _((old estimate)) −X _((old estimate)) +X_((measured))]

X _((new estimate)) =X _((measured))  Equation 3

[0039] In a real world application, the Gain is virtually always greaterthan 0 and less than 1. The Gain thus determines to what degree a newmeasurement can change the previous aggregate estimate or prediction ofthe location of an object, in the case of the instant invention, theoccupant 18 is the object being tracked. Both the shape tracker 48 andthe motion tracker 50 are described in greater detail below, along withFIGS. 9 and 10 respectively.

[0040] The output from both the shape tracker 48 and the motion tracker50 are used impact assessment subsystem at 52. In the preferredembodiment of the invention, the system 16 predicts the kinetic energyof the occupant 18 at the time that the airbag deployment system 36deploys the airbag. Kinetic energy is a function of both mass andvelocity.

Kinetic Energy=½*Mass*Velocity²  Equation 4

[0041] As described in greater detail below, mass of the occupant 18 iscalculated by using the volume of the occupant 18 and the density of theoccupant 18.

Mass=Volume*Density  Equation 5

[0042] The impact metric calculations are computed more frequently thanthe camera 22 can collect sensor readings. The impact assessmentsubsystem 52 is discussed in greater detail below.

[0043] The impact metric is sent by the impact assessment subsystem 52to the airbag controller 32 so that the airbag controller 32 cancommunicate the appropriate deployment strength to the airbag deploymentsystem 36.

D. Ellipse Fitting Subsystem

[0044]FIG. 4 illustrates the ellipse fitting routine implemented by theellipse fitting subsystem 44. In a preferred embodiment, the ellipsefitting subsystem 44 is software in the computer 30, but in alternativeembodiments, the ellipse fitting subsystem 44 could be housed in adifferent computer or device.

[0045] The upper ellipse 58 extends from the hips up to the head of theoccupant 18. The lower ellipse 56. extends down from the hips to includethe feet of the occupant 18. If the entire area from an occupant's 18hips down to the occupant's 18 feet is not visible, a lower ellipse isgenerated to represent what is visible. An ellipse can be tracked by thesystem 16 using a single point on the ellipse, preferably the centroidof the ellipse. In alternative embodiments, shapes other than ellipsescan be used to represent the upper and lower parts of an occupant 18.The ellipse fitting routine is known in the art. A preferred embodimentof the image processing system 16 does not utilize the lower ellipse 56.

[0046]FIG. 5 illustrates many of the characteristics that can beoutputted from the ellipse fitting subsystem 44 for use by the system16. A centroid 54 of the upper ellipse can be identified by the system16 for tracking characteristics of the occupant 18. It is known in theart how to identify the centroid 54 of an ellipse. Motioncharacteristics include the x-coordinate (“distance”) 60 of the centroid54 and a forward tilt angle (“θ”) 62. Shape measurements include they-coordinate (“height”) 58 of the centroid 54, the length of the majoraxis of the ellipse (“major”) 64 and the length of the minor axis of theellipse (“minor”) 66. Rate of change information, such as velocity andacceleration, are also captured for all shape and motion measurements,so in the preferred embodiment of the invention there are nine shapecharacteristics (height, height′, height″, major, major', major″, minor,minor', and minor″) and six motion characteristics (distance, distance′,distance″, θ,θ′, and θ″). The sideways tilt angle Φ is not shown becauseit is perpendicular to the image plane, and this the sideways titleangle Φ is derived, not measured, as discussed in greater detail below.Motion and shape characteristics are used to calculate the volume, andultimately the mass, of the occupant 18, so that the kinetic energy ofthe occupant 18 can be determined.

[0047]FIG. 6 illustrates the sideways tilt angle “(Φ”) 73. In thepreferred embodiment of the invention, there are only three shapestates, leaning left towards the driver (left) 76, sitting upright(center) 72, and leaning right away from the driver (right) 74, withtilt sideways tilt angles of −Φ, 0, and Φ. In a preferred embodiment, Φis set at a value between 15 and 40 degrees, depending on the nature ofthe vehicle being used.

E. Markov Probability Chains

[0048] The present invention is directed to a multiple-model probabilityweighted implementation of a first Kalman filter for all shapecharacteristics and a second Kalman filter for all motioncharacteristics. In a preferred embodiment, each shape characteristichas a separate Kalman filter equation for each shape state. Similarly,each motion characteristic has a separate Kalman filter equation foreach motion mode. In a preferred embodiment of the invention, theoccupant 18 has at least one shape state and at least one motion mode.There are certain predefined probabilities associated with a transitionfrom one state to another state. These probabilities can best beillustrated through the use of Markov chains.

[0049]FIG. 7 illustrates the three shape states used in the preferredembodiment of the invention. In the preferred embodiment, an occupant 18is either leaning towards the driver (“left”) 80, sitting upright(“center”) 78, or leaning away from the driver (“right”) 82. Theprobability of an occupant being in a particular state and then endingin a particular state can be identified by lines originating at aparticular shape state with arrows pointing towards the subsequent shapestate. For example, the probability of an occupant in center stateremaining in center state P^(C-C) is represented by the arrow at 84. Theprobability of moving from center to left P^(C-L) is represented by thearrow 92 and the probability of moving from center to right P^(C-R) is98. The total probabilities resulting from an initial state of center 78must add up to 1.

P ^(C-C) +P ^(C-L) +P ^(C-R)=1.0  Equation 6

[0050] Furthermore, all of the probabilities originating from anyparticular state must also add up to 1.0.

[0051] The arrow at 86 represents the probability that a left tiltingoccupant 18 will sit centered P^(L-C), by the next interval of time.Similarly, the arrow at 96 represents the probability that a lefttilting occupant will tilt right P^(L-R) by the next interval of time,and the arrow at 90 represents the probability that a left tiltingoccupant will remain tilting to the left P^(L-L). The sum of allpossible probabilities originating from an initial tilt state of leftmust equal 1.

P ^(L-C) +P ^(L-L) +P ^(L-R)=10  Equation 7

[0052] Lastly, the arrow at 100 represents the probability that a righttilting occupant will remain tilting to the right P^(R-R), the arrow at88 represents the probability that a right tilting occupant will enter acentered state P^(R-C), and the arrow at 94 represents the probabilitythat an occupant will tilt towards the left p^(R-L) The sum of allpossible probabilities originating from an initial tilt state of rightequals 1.

P ^(R-C) +P ^(R-L) +P ^(R-R)=1.0  Equation 8

[0053] As a practical matter, the typical video camera 22 used in thepreferred embodiment captures between 50 to 100 frames each second.Thus, it is essentially impossible for a left 80 leaning occupant tobecome a right 82 leaning occupant, or for a right 82 leaning occupantto become a left 80 leaning occupant, in a mere {fraction (1/50)} of asecond. It is far more likely that a left 80 leaning occupant will firstenter a center state 78 before becoming a right 82 leaning occupant, andsimilarly, it is far more realistic for a left 80 leaning occupant tobecome a centered 78 occupant before becoming a right 82 leaningoccupant. Thus, in the preferred embodiment of, P^(L-R) at 96 is alwaysset at zero and p^(R-L) at 94 will also always be set at zero. The threeprobability equations relating to shape state are thus as follows:

P ^(C-C) +P ^(C-L) +P ^(C-R)=10  Equation 9

P ^(R-C) +P ^(R-R)=1.0  Equation 10

P ^(L-C) +P ^(L-L)=1.0  Equation 11

[0054]FIG. 8 illustrates a similar Markov chain to represent therelevant probabilities relating to motion modes. The preferredembodiment of the invention uses three motion modes: stationary 102,represents a human occupant 18 in a mode of stillness, such as whileasleep; human 112, represents a occupant 18 behaving as a typicalpassenger in an automobile or other vehicle, one that is moving as amatter of course, but not in an extreme way; and crash 122, representsthe occupant 18 of a vehicle that is in a mode of crashing.

[0055] The probability of an occupant being in a particular state andthen ending in a particular state can be identified by lines originatingin the current state with arrows pointing to the new state. For example,the probability of an occupant in a stationary state remaining instationary state P^(S-S) is represented by the arrow at 108. Theprobability of moving from stationary to human P^(S-H) is represented bythe arrow 114 and the probability of moving from stationary to crashP^(S-C) is 90. The total probabilities resulting from an initial stateof stationary 102 must add up to 1.

P ^(S-S) +P ^(S-H) +P ^(S-C)=1.0  Equation 11

[0056] Similarly, the probability of human to human is P^(H-H) at 116,human to stationary is P^(H-S) at 112, human to crash is P^(H-C) at 120,and stationary to crash p^(H-C) is 90. The total probabilities resultingfrom an initial state of stationary 102 must add up to 1.

P ^(H-H) +P ^(H-C) +P ^(H-S)=1.0  Equation 12

[0057] Lastly, the probability of going from crash to crash is P^(C-C)at 124, crash to stationary is P^(C-S) at 110, and crash to human isP^(C-H) at 118. The total probabilities resulting from an initial stateof crash 122 must add up to 1.

P ^(C-C) +P ^(C-S) +P ^(C-H)=1.0  Equation 13

[0058] As a practical matter, it is highly unlikely (but not impossible)for an occupant 18 to ever leave the state of crash at 122 once thatstate has been entered. Under most scenarios, a crash at 122 ends thetrip for the occupant 18. Thus, in a preferred embodiment, P^(C-H) isset to nearly zero and P^(C-S) is also set to nearly zero. It isdesirable that the system 16 allow some chance of leaving a crash state122 or else the system 16 may get stuck in a crash state 122 in cases ofmomentary system 16 “noise” conditions or some other unusual phenomenon.Alternative embodiments can set p^(C-H) and P^(C-S) to any desirablevalue, including zero, or a probability substantially greater than zero.The three equations for motion mode probabilities in a preferredembodiment are as follows:

P ^(C-C) +P ^(C-H) +P ^(C-S)=1.0  Equation 14

P ^(H-H) +P ^(H-C) +P ^(H-S)=1.0  Equation 15

P ^(S-C) +P ^(S-H) +P ^(S-S)=1.0  Equation 16

[0059] The transition probabilities associated with the various shapestates and motion modes are used to generate a Kalman filter equationfor each combination of characteristic and state. The results of thosefilters can then be aggregated in to one result, using the variousprobabilities to give the appropriate weight to each Kalman filter. Allof the probabilities are predefined by the user of the invention.

[0060] The markov chain probabilities provide a means to weigh thevarious Kalman filters for each characteristic and for each state andeach mode. The tracking and predicting subsystem system 46 incorporatesthe markov chain probabilities in the form of two subsystems, the shapetracker and predictor 48 and the motion tracker and predictor 50.

F. Shape Tracker And Predictor

[0061]FIG. 9 discloses a detailed flow chart for the shape tracker andpredictor 48. In the preferred embodiment of the invention, the shapetracker and predictor 48 tracks and predicts the major axis of the upperellipse (“major”), the minor axis of the upper ellipse (“minor”), andthe y-coordinate of the centroid (“height”). Each characteristic has avector describing position, velocity, and acceleration information forthe particular characteristic. The major vector is [major, major′,major″], with major′ representing the rate of change in the major orvelocity and major″ representing the rate of change in major velocity oracceleration. Accordingly, the minor vector is [minor, minor′, minor″],and the height vector is [height, height′, height″]. Any other shapevectors will similarly have position, velocity, and accelerationcomponents. The first step in the shape tracking and prediction processis an update of the shape prediction at 126.

[0062] 1. Update Shape Prediction

[0063] An update shape prediction process is performed at 126. Thisprocess takes the last shape estimate and extrapolates that estimateinto a future prediction using a transition matrix.

Updated Vector Prediction=Transition Matrix*Last VectorEstimate  Equation 17

[0064] The transition matrix applies Newtonian mechanics to the lastvector estimate, projecting forward a prediction of where the occupant18 will be on the basis of its past position, velocity, andacceleration. The last vector estimate is produced at 130 as describedbelow. The process from 126 to 128, from 128 to 130, and from 130 to132, loops back to 126. The process at 126 requires that an estimate bepreviously generated at 130, so processing at 126 and 128 is not invokedthe first time through the repeating loop that is steps 126 through 132.

[0065] The following equation is then applied for all shape variablesand for all shape states, where x is the shape variable, Δ t representschange over time (velocity), and ½Δt² At represents acceleration.${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 18}:{U\quad p\quad d\quad a\quad t\quad e\quad d\quad V\quad e\quad c\quad t\quad o\quad r\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n}} = {\begin{pmatrix}1 & {\Delta \quad t} & {\frac{1}{2}\Delta \quad t^{2}} \\0 & 1 & {\Delta \quad t} \\0 & 0 & 1\end{pmatrix}*\begin{pmatrix}x \\x^{\prime} \\x^{''}\end{pmatrix}}$

[0066] In a preferred embodiment of the invention, there are nineupdated vector predictions at 134 because there are three shape statesand three non-derived shape variables in the preferred embodiment, and3×3=9. The updated shape vector predictions are:

[0067] Updated major for center state.

[0068] Updated major for right state.

[0069] Updated major for left state.

[0070] Updated minor for center state.

[0071] Updated minor for right state.

[0072] Updated minor for left state.

[0073] Updated height for center state.

[0074] Updated height for right state.

[0075] Updated height for left state.

[0076] 2. Update Covariance and Gain Matrices

[0077] After the shape predictions are updated for all variables and allstates at 126, the shape prediction covariance matrices, shape gainmatrices, and shape estimate covariance matrices must be updated at 128.The shape prediction covariance accounts for error in the predictionprocess. The gain, as described above, represents the weight that themost recent measurement is to receive and accounts for errors in themeasurement segmentation process. The shape estimate covariance accountsfor error in the estimation process.

[0078] The prediction covariance is updated first. The equation to beused to update each shape prediction covariance matrix is as follows:${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 19}:\begin{matrix}{S\quad h\quad a\quad p\quad e\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix}} = \begin{matrix}\lbrack {S\quad t\quad a\quad t\quad e\quad {Transition}\quad {Matrix}*{Old}}  \\{{Estimate}\quad {Covariance}\quad {Matrix}*} \\{ {{transpose}( {{State}\quad {Transition}\quad {Matrix}} )} \rbrack +} \\{{System}\quad {Noise}}\end{matrix}$

[0079] The state transition matrix is the matrix that embodies Newtonianmechanics used above to update the shape prediction. The old estimatecovariance matrix is generated from the previous loop at 130. On thefirst loop from 126 through 132, step 128 is skipped. Taking thetranspose of a matrix is simply the switching of rows with columns andcolumns with rows, and is known under the art. Thus, the transpose ofthe state transition matrix is the state transition matrix with the rowsas columns and the columns as rows. System noise is a matrix ofconstants used to incorporate the idea of noise in the system. Theconstants used in the system noise matrix are set by the user of theinvention, but the practice of selecting noise constants are known inthe art.

[0080] The next matrix to be updated is the gain matrix. As discussedabove, the gain represents the confidence of weight that a newmeasurement should be given. A gain of one indicates the most accurateof measurements, where past estimates may be ignored. A gain of zeroindicates the least accurate of measurements, where the most recentmeasurement is to be ignored and the user of the invention is to relysolely on the past estimate instead. The role played by gain isevidenced in the basic Kalman filter equation of Equation 1.

X _((new estimate)) =X ^((old estimate))+Gain[−X _((old estimate)) +X_((measured))]

[0081] The gain is not simply one number because one gain exists foreach combination of shape variable and shape state. The general equationfor updating the gain is Equation 20:${G\quad a\quad i\quad n} = \begin{matrix}{S\quad h\quad a\quad p\quad e\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x*} \\{t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad M\quad a\quad t\quad r\quad i\quad x} )}*} \\{i\quad n\quad {v( {R\quad e\quad s\quad i\quad d\quad u\quad e\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e} )}}\end{matrix}$

[0082] The shape covariance matrix is calculated above. The measurematrix is simply a way of isolating and extracting the positioncomponent of a shape vector while ignoring the velocity and accelerationcomponents for the purposes of determining the gain. The transpose ofthe measure matrix is simply [1 0 0]. The reason for isolating theposition component of a shape variable is because velocity andacceleration are actually derived components, only position can bemeasured by a snapshot. Gain is concerned with the weight that should beattributed to the actual measurement.

[0083] In the general representation of a Kalman filter,X_((new estimate))=X_((old estimate))+Gain[−X_((old estimate))+X_((measured))],the residue represents the difference between the old estimate and thenew measurement. There are entire matrices of residue covariances. Theinverse of the residue covariance matrix is used to update the gainmatrix. It is known in the art how to take the inverse of a matrix,which is a simple linear algebra process. The equation for residuecovariance matrix is Equation 21: $\begin{matrix}{{R\quad e\quad s\quad i\quad d\quad u\quad e}\quad} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e}\end{matrix} = \begin{matrix}\lbrack {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad n\quad t\quad M\quad a\quad t\quad r\quad i\quad x*P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e*}  \\{ {t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )}} \rbrack +} \\{M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad N\quad o\quad i\quad s\quad e}\end{matrix}$

[0084] The measurement matrix is a simple matrix used to isolate theposition component of a shape vector from the velocity and accelerationcomponents. The prediction covariance is calculated above. The transposeof the measurement matrix is simply a one row matrix of [1 0 0] insteadof a one column matrix with the same values. Measurement noise is aconstant used to incorporate error associated with the sensor 22 and thesegmentation process 40.

[0085] The last matrix to be updated is the shape estimate covariancematrix, which represents estimation error. As estimations are based oncurrent measurements and past predictions, the estimate error willgenerally be less substantial than prediction error. The equation forupdating the shape estimation covariance matrix is Equation 22:$\begin{matrix}{S\quad h\quad a\quad p\quad e\quad E\quad s\quad t\quad i\quad m\quad a\quad t\quad e} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix} = \begin{matrix}( {{I\quad d\quad e\quad n\quad t\quad i\quad t\quad y\quad M\quad a\quad t\quad r\quad i\quad x} - {G\quad a\quad i\quad n\quad M\quad a\quad t\quad r\quad i\quad x*}}  \\{ {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )*} \\{S\quad h\quad a\quad p\quad e\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad o\quad r\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix}$

[0086] An identity matrix is known in the art, and consists merely of adiagonal line of 1's going from top left to bottom right, with zeros atevery other location. The gain matrix is computed and described above.The measure matrix is also described above, and is used to isolate theposition component of a shape vector from the velocity and accelerationcomponents. The predictor covariance matrix is also computed anddescribed above.

[0087] 3. Update Shape Estimate

[0088] An update shape estimate process is invoked at 138. The firststep in this process is to compute the residue.

Residue=Measurement−(Measurement Matrix*Prediction Covariance)  Equation23

[0089] Then the shape states themselves are updated.

Updated Shape Vector Estimate=Shape VectorPrediction+(Gain*Residue)  Equation 24

[0090] When broken down into individual equations, the results are asfollows:

X ^(C) _((major at t)) =X ^(C) _((major at t))+Gain[−X ^(C)_((major at t−1)) +X ^(C) _((measured major))]

X ^(L) _((major at t)) =X ^(L) _((major at t))+Gain[−X ^(L)_((major at t−1)) +X ^(L) _((measured major))]

X ^(R) _((major at t)) =X ^(R) _((major at t))+Gain[−X ^(R)_((major at t−1)) +X ^(R) _((measured major))]

X ^(C) _((minor at t)) =X ^(C) _((minor at t))+Gain[−X ^(C)_((minor at t−1)) +X ^(C) _((measured minor))]

X ^(L) _((minor at t)) =X ^(L) _((major at t))+Gain[−X ^(L)_((minor at t−1)) +X ^(L) _((measured minor))]

X ^(R) _((minor at t)) =X ^(R) _((minor at t))+Gain[−X ^(R)_((minor at t−1)) +X ^(R) _((measured minor))]

X ^(C) _((height at t)) =X ^(C) _((height at t))+Gain[−X ^(C)_((height at t−1)) +X ^(C) _((measured height))]

X ^(L) _((height at t)) =X ^(L) _((height at t))+Gain[−X ^(L)_((height at t−1)) +X ^(L) _((measured height))]

X ^(R) _((height at t)) =X ^(R) _((height at t))+Gain[−X ^(R)_((height at t−1)) +X ^(R) _((measured height))]

[0091] In the preferred embodiment, C represents the state of center, Lrepresents the state of leaning left towards the driver, and Rrepresents the state of leaning right away from the driver.

[0092] 4. Generate Combined Shape Estimate

[0093] The last step in the repeating loop between steps 126 and steps132 is a generate combined shape estimate step at 132. The first part ofthat process is to assign a probability to each shape vector estimate.The residue covariance is re-calculated, using the same formula asdiscussed above.${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 25}:\begin{matrix}{{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad R\quad e\quad s\quad i\quad d\quad u\quad e}\quad} \\{M\quad a\quad t\quad r\quad i\quad x}\end{matrix}} = \begin{matrix}\lbrack {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x*}  \\{P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x*} \\{ {t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )}} \rbrack +} \\{M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad N\quad o\quad i\quad s\quad e}\end{matrix}$

[0094] Next, the actual likelihood for each shape vector is calculated.The system 16 determines which state the occupant is in by comparing thepredicted values for the various states with the recent best estimate ofwhat the current values for the shape variables actually are.${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 26}:{L\quad i\quad k\quad e\quad l\quad i\quad h\quad o\quad o\quad {d\begin{pmatrix}C \\R \\L\end{pmatrix}}}} = ^{{{- {({{residue} - {offset}})}^{2}}/2}\sigma^{2}}$

[0095] There is no offset in the preferred embodiment of the inventionbecause it is assumed that offsets cancel each other out in theprocessing performed by the system 16. Sigma represents variance, and isdefined in the implementation phase of the invention by a humandeveloper. It is known in the art how to assign a useful value for sigmaby looking at data.

[0096] The state with the highest likelihood determines the sidewaystilt angle Φ. If the occupant 18 is in a centered state, the sidewaystilt angle is 0 degrees. If the occupant 18 is tilting left, then thesideways tilt angle is −Φ. If the occupant 18 is tilting towards theright, the sideways tilt angle is Φ. In the preferred embodiment of theinvention, Φ and −Φ are predefined on the basis of the type and model ofvehicle using the system 16.

[0097] Next, state probabilities are updated from the likelihoodgenerated above and the pre-defined markovian mode probabilitiesdiscussed above.

P ^(C) =P ^(C-C) +P ^(R-C) +P ^(L-C)  Equation 27

P ^(R) =P ^(R-R) +P ^(C-R)  Equation 28

P ^(L) =P ^(L-L) +P ^(C-L)  Equation 27

[0098] The equations for the updated mode probabilities are as follows,where L represents the likelihood of a particular mode as calculatedabove:${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 30}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad L\quad e\quad f\quad t}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{L}*( {P^{L - L} + P^{C - L}} )} + {L^{R}*( {P^{R - R} +} }} } \\{ { P^{C - R} ) + {L^{C}*( {P^{C - C} + P^{R - C} + P^{L - C}} )}} \rbrack*} \\{L^{L}*( {P^{L - L} + P^{C - L}} )}\end{matrix}$${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 31}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad R\quad i\quad g\quad h\quad t}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{L}*( {P^{L - L} + P^{C - L}} )} + {L^{R}*( {P^{R - R} +} }} } \\{ { P^{C - R} ) + {L^{C}*( {P^{C - C} + P^{R - C} + P^{L - C}} )}} \rbrack*} \\{L^{R}*( {P^{R - R} + P^{C - R}} )}\end{matrix}$${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 32}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad C\quad e\quad n\quad t\quad e\quad r}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{L}*( {P^{L - L} + P^{C - L}} )} + {L^{R}*( {P^{R - R} +} }} } \\{ { P^{C - R} ) + {L^{C}*( {P^{C - C} + P^{R - C} + P^{L - C}} )}} \rbrack*} \\{L^{C}*( {P^{C - C} + P^{L - C} + P^{L - C}} )}\end{matrix}$

[0099] The combined shape estimate is ultimately calculated by usingeach of the above probabilities, in conjunction with the various shapevector estimates.${E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 33}:{\begin{matrix}{X = \quad {{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad L\quad e\quad f\quad t*X^{L\quad e\quad f\quad t}} +}} \\{\quad {{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad R\quad i\quad g\quad h\quad t*X^{R\quad i\quad g\quad h\quad t}} +}} \\{\quad {P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad C\quad e\quad n\quad t\quad e\quad r*X^{C\quad e\quad n\quad t\quad e\quad r}}}\end{matrix}\quad}$

[0100] X is any of the shape variables, including a velocity oracceleration derivation of a measure value.

[0101] The loop from 126 through 132 repeats continuously while thevehicle is in operation or while there is an occupant 18 in the seat 20.The process at 126 requires that an estimate be previously generated at130, so processing at 134 and 136 is not invoked the first time throughthe repeating loop that is steps 136-132.

G. Motion Tracker and Predictor

[0102] The motion tracker and predictor 50 in FIG. 10 functionssimilarly in many respects, to the shape tracker and predictor 48 inFIG. 9. The motion tracker and predictor 50 tracks differentcharacteristics and vectors than the shape tracker. In the preferredembodiment of the invention, the x-coordinate of the centroid and theforward tilt angle θ (“θ”), and their corresponding velocities andaccelerations (collectively “motion variables”) are tracked andpredicted. The x-coordinate of the centroid is used to determine thedistance between the occupant 18 and a location within the automobilesuch as the instrument panel 34, the airbag deployment system 36, orsome other location in the automobile. In the preferred embodiment, theinstrument panel 34 is used since that is where the airbag is generallydeployed from.

[0103] The x-coordinate vector includes a position component (x), avelocity component (x′), and an acceleration component (x″). The θvector similarly includes a position component (θ), a velocity component(θ′), and an acceleration component (θ″). Any other motion vectors willsimilarly have position, velocity, and acceleration components.

[0104] 1. Update Motion Prediction

[0105] An update motion prediction process is performed at 134. Thisprocess takes the last motion estimate and extrapolates that estimateinto a future prediction using a transition matrix as disclosed inEquation 17:.

Updated Vector Prediction=Transition Matrix*Last Vector Estimate

[0106] The transition matrix applies Newtonian mechanics to the lastvector estimate, projecting forward a prediction of where the occupant18 will be on the basis of its past position, velocity, andacceleration. The last vector estimate is produced at 138 as describedbelow. The process from 134 to 136, from 136 to 138, and from 138 to140, loops back to 134. The process at 134 requires that an estimate bepreviously generated at 138, so processing at 134 and 136 is not invokedthe first time through the repeating loop that is steps 134-140.

[0107] Equation 18 is then applied for all motion variables and for allmotion modes:${U\quad p\quad d\quad a\quad t\quad e\quad d\quad V\quad e\quad c\quad t\quad o\quad r\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n} = {\begin{pmatrix}1 & {\Delta \quad t} & {\frac{1}{2}\Delta \quad t^{2}} \\0 & 1 & {\Delta \quad t} \\0 & 0 & 1\end{pmatrix}*\begin{pmatrix}x \\x^{\prime} \\x^{''}\end{pmatrix}}$

[0108] In the preferred embodiment of the invention, there would be sixupdated vector predictions at 134 because there are three motion modesand two motion variables in the preferred embodiment, and 3×2=6. Theupdated motion predictions are:

[0109] Updated x-coordinate for crash mode.

[0110] Updated x-coordinate for human mode.

[0111] Updated x-coordinate for stationary mode.

[0112] Updated θ for crash mode.

[0113] Updated θ for human mode.

[0114] Updated θ for stationary mode.

[0115] 2. Update Covariance and Gain Matrices

[0116] After the motion predictions are updated for all motion variablesand all modes at 134, the motion prediction covariance matrices, motiongain matrices, and motion estimate covariance matrices must be updatedat 136. The motion prediction covariance accounts for error in theprediction process. The gain, as described above, represents the weightthat the most recent measurement is to receive and accounts for errorsin the measurement and segmentation process. The motion estimatecovariance accounts for error in the estimation process.

[0117] The prediction covariance is updated first. Equation 34 is usedto update each motion prediction covariance matrix.${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 34}:\begin{matrix}{M\quad o\quad t\quad i\quad o\quad n\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix}} = \begin{matrix}{S\quad t\quad a\quad t\quad e\quad {Transition}\quad {Matrix}*{Old}} \\{{Estimate}\quad {Covariance}\quad {Matrix}*} \\{{{transpose}( {{State}\quad {Transition}\quad {Matrix}} )} +} \\{{System}\quad {Noise}}\end{matrix}$

[0118] The state transition matrix is the matrix that embodies Newtonianmechanics used above to update the motion prediction. The old estimatecovariance matrix is generated from the previous loop at 136. On thefirst loop from 134 through 140, step 134 is skipped. Taking thetranspose of a matrix is simply the switching of rows with columns andcolumns with rows, and is known under the art. Thus, the transpose ofthe state transition matrix is the state transition matrix with the rowsas columns and the columns as rows. System noise is a matrix ofconstants used to incorporate the idea of noise in the system. Theconstants used in the system noise matrix are set by the user of theinvention, but the practice of selecting such constants are known in theart.

[0119] The next matrix to be updated is the gain matrix. As discussedabove, the gain represents the confidence of weight that a newmeasurement should be given. A gain of one indicates the most accurateof measurements, where past estimates may be ignored. A gain of zeroindicates the least accurate of measurements, where the most recentmeasurement is to be ignored and the user of the invention is to rely onthe past estimate instead. The role played by gain is evidenced in thebasic Kalman filter equation in Equation 1 where

X _((new estimate)) =X _((old estimate))+Gain[−X _((old estimate)) +X_((measured))]

[0120] The gain is not simply one number but an entire matrix becauseone gain exists for each combination of motion variable and motion mode.The general equation for updating the gain is Equation 35:${G\quad a\quad i\quad n} = \begin{matrix}{M\quad o\quad t\quad i\quad o\quad n\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x*} \\{t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad M\quad a\quad t\quad r\quad i\quad x} )}*} \\{i\quad n\quad {v( {R\quad e\quad s\quad i\quad d\quad u\quad e\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e} )}}\end{matrix}$

[0121] The motion covariance matrix is calculated above. The measurematrix is simply a way of isolating and extracting the positioncomponent of a motion vector while ignoring the velocity andacceleration components for the purposes of determining the gain. Thetranspose of the measure matrix is simply [1 0 0]. The reason forisolating the position component of a motion variable is becausevelocity and acceleration are actually derived components. Position isthe only component actually measured, and because gain is concerned withthe weight that should be attributed to the actual measurement, derivedvariables should be isolated.

[0122] In the general representation of a Kalman filter,X_((new estimate))=X_((old estimate))+Gain[−X_((old estimate))+X_((measured))],the residue represents the difference between the old estimate and thenew measurement. There are entire matrices of residue covariances. Theinverse of the residue covariance matrix is used to update the gainmatrix. It is known in the art how to take the inverse of a matrix,which is a simple linear algebra process. The equation for residuecovariance matrix is Equation 21 as disclosed above: $\begin{matrix}{{R\quad e\quad s\quad i\quad d\quad u\quad e}\quad} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e}\end{matrix} = \begin{matrix}\lbrack {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad n\quad t\quad M\quad a\quad t\quad r\quad i\quad x*P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e*}  \\{ {t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )}} \rbrack +} \\{M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad N\quad o\quad i\quad s\quad e}\end{matrix}$

[0123] The measurement matrix is a simple matrix used to isolate theposition component of a motion vector from the velocity and accelerationcomponents. The prediction covariance is calculated above. The transposeof the measurement matrix is simply a one row matrix of [1 0 0] insteadof a one column matrix with the same values. Measurement noise is aconstant used to incorporate error associated with the sensor 22 and thesegmentation process 40.

[0124] The last matrix to be updated is the motion estimate covariancematrix, which represents estimation error. As estimations are based oncurrent measurements and past predictions, the estimate error willgenerally be less substantial than the prediction error. The equationfor updating the motion estimation covariance matrix is Equation 36:$\begin{matrix}{M\quad o\quad t\quad i\quad o\quad n\quad E\quad s\quad t\quad i\quad m\quad a\quad t\quad e} \\{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix} = \begin{matrix}( {{I\quad d\quad e\quad n\quad t\quad i\quad t\quad y\quad M\quad a\quad t\quad r\quad i\quad x} - {G\quad a\quad i\quad n\quad M\quad a\quad t\quad r\quad i\quad x*}}  \\{ {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )*} \\{M\quad o\quad t\quad i\quad o\quad n\quad P\quad r\quad e\quad d\quad i\quad c\quad t\quad o\quad r\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x}\end{matrix}$

[0125] An identity matrix is known in the art, and consists merely of adiagonal line of 1's going from top left to bottom right, with zeros atevery other location. The gain matrix is computed and described above.The measure matrix is also described above, and is used to isolate theposition component of a motion vector from the velocity and accelerationcomponents. The predictor covariance matrix is also computed anddescribed above.

[0126] 3. Update Motion Estimate

[0127] An update motion estimate process is invoked at 138. The firststep in this process is to compute the residue using Equation 23:${{Re}\quad s\quad i\quad d\quad u\quad e} = \begin{matrix}{{M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}} -} \\( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x*}  \\ {P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e} )\end{matrix}$

 Motion Vector Estimate=Motion VectorPrediction+(Gain*Residue)  Equation 37

[0128] When broken down into individual equations, the results are asfollows:

X ^(H) _((x-coordinate at t)) =X ^(H) _((x-coordinate at t))+Gain[−X^(H) _((x-coordinate at t−1)) +X ^(H) _((measured x-coordinate))]

X ^(S) _((x-coordinate at t)) =X ^(S) _((x-coordinate at t))+Gain[−X^(S) _((x-coordinate at t−1)) +X ^(S) _((measured x-coordinate))]

X ^(C) _((x-coordinate at t)) =X ^(C) _((x-coordinate at t))+Gain[−X^(C) _((x-coordinate at t−1)) +X ^(C) _((measured x-coordinate))]

X ^(H) _((θ at t)) =X ^(H) _((θ at t))+Gain[−X ^(H) _((θ at t−1)) +X^(H) _((measured θ))]

X ^(S) _((θ at t)) =X ^(S) _((θ at t))+Gain[−X ^(S) _((θ at t−1)) +X^(S) _((measured θ))]

X ^(C) _((θ at t)) =X ^(C) _((θ at t−1))+Gain[−X ^(C) _((θ at t−1)) +X^(C) _((measured θ))]

[0129] In the preferred embodiment, H represents the mode of human, Crepresents the mode of crash, and S represents the mode of stationary.

[0130] 4. Generate Combined Motion Estimate

[0131] The last step in the repeating loop between steps 134 and steps140 is a generate combined motion estimate step at 140. The first partof that process is to assign a probability to each motion vectorestimate. The residue covariance is re-calculated, using Equation 25 asdiscussed above. $\begin{matrix}{{C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad R\quad e\quad s\quad i\quad d\quad u\quad e}\quad} \\{M\quad a\quad t\quad r\quad i\quad x}\end{matrix} = \begin{matrix}\lbrack {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x*}  \\{P\quad r\quad e\quad d\quad i\quad c\quad t\quad i\quad o\quad n\quad C\quad o\quad v\quad a\quad r\quad i\quad a\quad n\quad c\quad e\quad M\quad a\quad t\quad r\quad i\quad x*} \\{ {t\quad r\quad a\quad n\quad s\quad p\quad o\quad s\quad {e( {M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad M\quad a\quad t\quad r\quad i\quad x} )}} \rbrack +} \\{M\quad e\quad a\quad s\quad u\quad r\quad e\quad m\quad e\quad {nt}\quad N\quad o\quad i\quad s\quad e}\end{matrix}$

[0132] Next, the actual likelihood for each motion vector is calculated.${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 38}:{L\quad i\quad k\quad e\quad l\quad i\quad h\quad o\quad o\quad {d\begin{pmatrix}C \\H \\S\end{pmatrix}}}} = ^{{{- {({{residue} - {offset}})}^{2}}/2}\sigma^{2}}$

[0133] There is no offset in a preferred embodiment of the inventionbecause it can be assumed that offsets cancel each other out, and thatthe system's 16 processes can be zero-mean Gaussian signals. Sigmarepresents variance, is defined in the implementation phase of theinvention by a human developer. It is known in the art how to assign auseful value for sigma by looking at data.

[0134] Next, mode probabilities are updated from the likelihoodgenerated above and the pre-defined markovian mode probabilitiesdiscussed above.

P ^(C) =P ^(C-C) +P ^(S-C) +P ^(H-C)  Equation 39

P ^(H) =P ^(H-H) +P ^(S-H) +P ^(C-H)  Equation 40

P ^(S) =P ^(S-S) +P ^(H-S) +P ^(C-S)  Equation 41

[0135] The equations for the updated mode probabilities are as follows,where L represents the likelihood of a particular mode as calculatedabove:${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 42}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad {Stationary}}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{S}*( {P^{S - S} + P^{H - S} + P^{C - S}} )} + {L^{H}*}} } \\{( {P^{H - H} + P^{S - H} + P^{C - H}} ) + {L^{C}*( {P^{C - C} +} }} \\{  {P^{S - C} + P^{H - C}} ) \rbrack*L^{S}*( {P^{S - S} + P^{H - S} +} } \\ P^{C - S} )\end{matrix}$${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 43}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad {Human}}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{S}*( {P^{S - S} + P^{H - S} + P^{C - S}} )} + {L^{H}*}} } \\{( {P^{H - H} + P^{S - H} + P^{C - H}} ) + {L^{C}*( {P^{C - C} +} }} \\{  {P^{S - C} + P^{H - C}} ) \rbrack*L^{H}*( {P^{H - H} + P^{S - H} +} } \\ P^{C - H} )\end{matrix}$${{E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 44}:\begin{matrix}{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f} \\{m\quad o\quad d\quad e\quad C\quad {rash}}\end{matrix}} = \begin{matrix}{1/\lbrack {{L^{S}*( {P^{S - S} + P^{H - S} + P^{C - S}} )} + {L^{H}*}} } \\{( {P^{H - H} + P^{S - H} + P^{C - H}} ) + {L^{C}*( {P^{C - C} +} }} \\{  {P^{S - C} + P^{H - C}} ) \rbrack*L^{C}*( {P^{C - C} + P^{S - C} +} } \\ P^{H - C} )\end{matrix}$

[0136] The combined motion estimate is ultimately calculated by usingeach of the above probabilities, in conjunction with the various motionvector estimates.${E\quad q\quad u\quad a\quad t\quad i\quad o\quad n\quad 45}:{\begin{matrix}{X = \quad {{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad H\quad u\quad m\quad a\quad n*X^{H\quad u\quad m\quad a\quad n}} +}} \\{\quad {{P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad C\quad r\quad a\quad s\quad h*X^{C\quad r\quad a\quad s\quad h}} +}} \\{\quad {P\quad r\quad o\quad b\quad a\quad b\quad i\quad l\quad i\quad t\quad y\quad o\quad f\quad m\quad o\quad d\quad e\quad S\quad t\quad a\quad t\quad i\quad o\quad n\quad a\quad r\quad y*X^{S\quad t\quad a\quad t\quad i\quad o\quad n\quad a\quad r\quad y}}}\end{matrix}\quad}$

[0137] X is any of the motion variables, including a velocity oracceleration derivation.

[0138] The loop from 126 through 132 repeats continuously while thevehicle is in operation or while there is an occupant 18 in the seat 20.

H. Impact Assessment Subsystem

[0139] The impact assessment subsystem 52 uses the shape and motionvariables above to generate a impact metric representing the occupant 18impact that an airbag needs to absorb. In a preferred embodiment,kinetic energy is the impact metric. In alternative embodiments,momentum, or a weighted combination of kinetic energy and momentum canbe used as the impact metric. Alternative embodiments can utilize anyimpact metric incorporating the characteristics of mass, velocity, orany of the other motion or shape variables, including anycharacteristics that could be derived from a motion or shape variable.

[0140] 1. Mass

[0141] As disclosed in Equation 4, mass is used to compute the impactmetric. The density of a human occupant 18 is relatively constant acrossbroad spectrum of potential human occupants 18. The average density of ahuman occupant 18 is known in the art as anthropomorphic data that canbe obtained from NHTSA (National Highway Traffic Safety Administration)or the IIA (Insurance Institute of America). The mass of an occupant 18is substantially a function of volume.

Mass=Volume*Density  Equation 46

[0142] In a preferred embodiment, the system 16 determines whether ornot the occupant 18 is restrained by a seat belt. This is done in bycomparing the velocity (x′) of the occupant 18 with the rate of changein the forward tilt angle (θ′). If the occupant is restrained by a seatbelt, the rate of change in the forward tilt angle should be roughly twotimes the velocity of the occupant. In contrast, for an unbeltedoccupant, the ratio of θ′/x′ will be roughly zero, because there will bean insignificant change in the forward tilt angle for an unbeltedoccupant. If an occupant 18 is restrained by a functional seatbelt, themass of the occupant's 18 lower torso should not be included in theimpact metric of the occupant 18 because the mass of the lower torso isrestrained by a seal belt, and thus will not need to be constrained bythe airbag deployment system 36. If the occupant 18 is not restrained bya seatbelt, the mass of the lower torso needs to be included in the massof the occupant 18. Across the broad spectrum of potential humanoccupants 18, the upper torso is consistently between 65% and 68% of thetotal mass of a human occupant 18. If the occupant 18 is not restrainedby a seat belt in a preferred embodiment, the mass of both the occupant18 (including the lower torso) is calculated by taking the mass of theupper torso and dividing that mass by a number between 0.65 and 0.68. Apreferred embodiment does not require the direct calculation of thevolume or mass of the lower ellipse 56.

[0143] The volume of an ellipsoid is well known in the art.

Volume=4/3*π*major*minor₁*minor₂  Equation 47

[0144] Major is the major axis 64. Minor₁ is the minor axis 66. Minor₂is the minor axis that is not visible in FIG. 5 because it is in thez-axis of FIG. 5, and represents the “width” of the ellipsoid, or theshoulder-span of the occupant 18. The 2-D ellipse is known to be aprojection from a particular angle and therefore allows the system 16 todecide what the originating 3-D Ellipsoid should be. In a preferredembodiment, the “width” of the ellipsoid is capped at the width of thevehicle seat 20 in which the occupant 18 sits. The width of the vehicleseat 20 can be easily measured for any vehicle before the system 16 isused for a particular vehicle model or type.

[0145] Minor₂ is derived from the major axis 64 and the minor axis 66.Anthropomorphic data from NHTSA or the Insurance Institute of America isused to create electronic “look-up” tables deriving the z-axisinformation from the major axis 64 and minor axis 66 values. FIG. 11aillustrates the format of a “look-up” table that could be electronicallystored in the computer 30. The inputs of a major axis value 64 and aminor axis value 66 are used to derive the z-axis value, which can thenbe used to calculate volume using Equation 47, and then mass usingEquation 46.

[0146]FIG. 11b illustrates a “look-up” table where Equation 47 isincorporated directly into output listing. Inputting the major 64 andminor 66 axis values results in the output of volume rather than merelythe z-axis value because the operations of Equation 47 are alreadyperformed on the z-axis value.

[0147]FIG. 11c illustrates a preferred embodiment of the “look-up” tablewhere both Equations 47 and 48 are incorporated into the output column,and thus the system inputs the major 64 and minor 66 axis values, whileoutputting the mass of the occupant 18.

[0148] 2. Velocity

[0149] Velocity is a motion characteristic derived from the differencesin occupant 18 position as described by Newtonian mechanics and isdescribed in greater detail above. The relevant measure of occupant 18velocity is the moment of impact between the occupant 18 and the airbag.The movement of the airbag towards the occupant 18 is factored into thisanalysis in the preferred embodiment of the system 16.

∫Velocity_(occupant) δt=∫Velocity_(airbag) δt  Equation 49

[0150] 3. Additional Alternative Variations and Embodiments

[0151] The underlying calculations of motion and shape variables areupdated very quickly using the outputted state transition matrix whichallows the system 16 to predict the position and shape in advance, andat a rate more quickly than the rate in which the sensor 22 collectsdata. The impact metric prediction is thus similarly updated at aquicker rate than the rate in which the sensor 22 collects data. Inalternative embodiments of the invention that classify the occupant 18into different occupant types, each occupant type could have a distinctdensity.

[0152] Other characteristics such as momentum can be substituted forkinetic energy as a proxy for the desirable strength of an airbagdeployment. Other potential impact metrics include momentum, a weightedcombination of momentum and kinetic energy, or potentially any othermetric utilizing mass or velocity. Regardless of the particular impactmetric used, the impact metric should be updated at a quicker rate thanthe rate in which the sensor 22 collects data to avoid the need forusing an expensive high-speed camera.

[0153] In accordance with the provisions of the patent statutes, theprinciples and modes of operation of this invention have been explainedand illustrated in preferred embodiments. However, it must be understoodthat this invention may be practiced otherwise than is specificallyexplained and illustrated without departing from its spirit or scope.

What is claimed is:
 1. An image processing system for use with an airbagdeployment system having a seat, an occupant in the seat, a sensor forgenerating sensor readings, an airbag, and an airbag controller, saidimage processing system comprising: a tracking and predicting subsystem,including a sensor reading and an occupant characteristic, said trackingand predicting subsystem generating said occupant characteristic fromsaid sensor reading; and an impact assessment subsystem, including aimpact metric, said impact assessment subsystem generating said impactmetric from said occupant characteristic.
 2. An image processing systemas recited in claim 1, wherein a kinetic energy prediction is saidimpact metric.
 3. An image processing system as recited in claim 1,wherein said impact metric relates to a point in time when the occupantimpacts into the deploying airbag.
 4. An image processing system asrecited in claim 1, wherein said impact metric is generated at a quickerrate than the rate at which the sensor generates said sensor readings.5. An image processing system as recited in claim 1, further comprisingan ellipse fitting subsystem and an ellipse, said ellipse fittingsubsystem generating said ellipse to represent the occupant.
 6. An imageprocessing system as recited in claim 1, wherein said tracking andpredicting subsystem is an iterative tracking and predicting subsystem.7. An image processing system as recited in claim 6, said iterativetracking and predicting subsystem including a plurality of Kalmanfilters, wherein said Kalman filters generate said impact metric.
 8. Animage processing system as recited in claim 1, wherein said impactmetric determines the deployment strength of the airbag.
 9. An imageprocessing system as recited in claim 1, said impact assessmentsubsystem including a seat belt status.
 10. An image processing systemas recited in claim 1, further including a plurality of occupantcharacteristics comprising: a velocity; and one said occupantcharacteristic selected from the group including a mass; a volume; anarea; and an axis of a three-dimensional environment.
 11. An imageprocessing system as in claim 1, said tracking and predicting subsystemfurther including: a plurality of occupant characteristics, comprisingshape characteristics and motion characteristics; a motion tracker andpredictor for tracking and predicting said motion characteristics; and ashape tracker and predictor for tracking and predicting said shapecharacteristics. shape characteristics; wherein said impact metric isbased on at least one said motion characteristic and at least one saidshape characteristic.
 12. An image processing system as in claim 11,wherein said shape characteristics include: a height of the occupant;and a shape of the upper torso of the occupant.
 13. An image processingsystem as in claim 11, wherein said motion characteristics include: adistance between the occupant and the airbag; and a forward tilt angle.14. An image processing system as in claim 11, including a plurality ofpredefined shape states, wherein said shape tracker and predictor usessaid plurality of predefined shape states.
 15. An image processingsystem as in claim 14, wherein said predefined states comprise a stateof leaning left; a state of leaning right; and a state of sittinggenerally centered.
 16. An image processing system as in claim 11,including a plurality of predefined motion modes, wherein said motiontracker and predictor utilizes said plurality of predefined motionmodes.
 17. An image processing system as in claim 16, wherein saidpredefined motion modes comprise: a mode of crashing; a mode of beingstationary; and a mode of being human.
 18. An image classificationsystem as in claim 11, wherein said shape tracker and predictorcomprises: an update shape predictor; an update covariance and gainmatrices generator; an update shape estimator; and a combined shapeestimate generator.
 19. An image classification system as in claim 11,wherein said motion tracker and predictor comprises: an update motionpredictor; an update covariance and gain matrices generator; an updatemotion estimator; and a combined motion estimate generator.
 20. An imageprocessing system as in claim 11, wherein said shape tracker andpredictor determines a sideways tilt angle of the occupant.
 21. An imageprocessing system for use with an airbag deployment system having aseat, an occupant in the seat, a sensor for capturing occupant images,an airbag, an airbag controller, said image processing systemcomprising: a segmentation subsystem, including an ambient image and asegmented image, said segmentation subsystem generating said segmentedimage from said ambient image; an ellipse fitting subsystem, includingan ellipse, said ellipse fitting subsystem representing said ambientimage with said ellipse; a tracking and predicting subsystem, includinga plurality of occupant characteristics, said tracking an predictingsubsystem generating said plurality of occupant characteristics fromsaid ellipse and an impact assessment subsystem, including an impactmetric, said impact assessment subsystem generating said impact metricfrom said plurality of occupant characteristics.
 22. An image processingsystem as recited in claim 21, said tracking and predicting subsystemfurther including a plurality of past predictions, wherein saidplurality of past predictions are incorporated into said plurality ofoccupant characteristics.
 23. An image processing system as recited inclaim 22, said tracking and predicting subsystem applying a plurality ofKalman filter to incorporate said plurality of past predictions intosaid plurality of occupant characteristics.
 24. A method for determiningairbag deployment strength, comprising the steps of: applying aplurality of mathematical heuristics to a plurality of imagecharacteristics to incorporate past measurements and past predictionsinto a plurality of updated occupant characteristic predictions, andcalculating an impact metric representing the magnitude of the impactbetween the occupant and the airbag from the updated occupantcharacteristic predictions.
 25. A method for determining airbagdeployment strength as recited in claim 24, wherein the plurality ofmathematical heuristics are Kalman filters.
 26. A method for determiningairbag deployment strength as recited in claim 24, wherein the impactmetric is the kinetic energy of the occupant.